Stereo imaging system and method for use in telerobotic systems

ABSTRACT

This invention relates to a stereo imaging system for use in telerobotic systems. A method of imaging a target site in a stereo imaging system is provided. The method typically includes capturing a right and a left optical image of the target site and transforming the right and the left optical images preferably into digital information. The method further includes converting the digital information into opposed images of the target site displayed on a stereo display of the stereo imaging system, one of the opposed images being associated with the right optical image and the other of the opposed images being associated with the left optical image. The method further includes regulating the digital information to cause the positions of the target site displayed on the opposed images to change relative to each other. The invention further provides for a method of aligning the opposed images, to a method of adjusting the stereo working distance of an image capture device, such as an endoscope and to a stereo imaging system.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority from, U.S. ProvisionalPatent Application Ser. No. 60/111,714, filed Dec. 8, 1998, the fulldisclosure of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

This invention relates to stereo imaging systems, devices, and methods.In particular, the invention relates to a method of imaging a targetsite with a stereo imaging system, a method of aligning images of astereo imaging system, a method of adjusting the stereo working distanceof a stereo imaging system, and a stereo imaging system.

In minimally invasive surgery such as laparoscopic surgery, for example,a patient's abdomen is insufflated with gas, and trocar sleeves orcannulas are passed through small incisions to provide entry ports forlaparoscopic surgical instruments. The laparoscopic surgical instrumentsgenerally include an endoscope in the form of a laparoscope for viewingthe surgical site or field and surgical instruments defining endeffectors such as clamps, graspers, scissors, staplers, and needleholders. The surgical instruments are similar to those used inconventional (open) surgery, except that the working end of each tool isseparated from its handle by an elongate shaft. To perform surgicalprocedures, the surgeon passes instruments through the trocar sleevesand manipulates them from outside the abdomen by sliding them in and outthrough the sleeves in the abdominal wall, and actuating end effectorson distal ends of the instruments, while viewing the surgical sitethrough the laparoscope.

In robotically-assisted and telerobotic surgery (both open andendoscopic procedures), the position of the surgical instruments istypically controlled by servo motors rather than directly by hand. Theservo motors follow the motions of a surgeon's hands as he or shemanipulates input or master control devices whilst remotely viewing theoperation via an image displayed on a viewer, the viewer beingoperatively linked to an image capture device typically in the form ofan endoscope. The viewer and its associated image capture device formpart of an imaging system. The servo motors are typically part of anelectromechanical surgical apparatus, which typically includes roboticarms that support and control the surgical instruments that have beenintroduced into e.g. an open surgical site, or through trocar sleevesinto a body cavity, such as the patient's abdomen, or the like. Duringthe operation, the master control devices provide mechanical actuationand control of a variety of surgical instruments. Such surgicalinstruments can typically include tissue graspers, needle drivers, etc.,that can perform various surgical procedures for the surgeon, i.e.,holding or driving a needle, grasping a blood vessel, dissecting tissue,and the like, while the surgeon views the procedures on the viewer.

It will be appreciated that the imaging system should meet certaincriteria to render it suitable for use in applications such as minimallyinvasive surgical applications. In particular, the image displayed tothe surgeon should be clear and optically correct. Furthermore, theimaging system should provide an adequate field of view and adequateresolution and brightness, so as to enable the surgical field to beadequately visualized by the surgeon.

Stereo endoscopes are sometimes used to provide the surgeon with astereo image at the viewer. Stereo endoscopes are typically arranged tohave a fixed point of intersection between two viewing axes. Thedistance between the fixed point and an object viewing end of theendoscope is referred to as the “working distance” of the endoscope. Inuse, the surgeon, whilst performing a surgical procedure, may want toobserve an object removed from the point of intersection. For example,when using the endoscope to “search” for the surgical site, the surgeontypically observes objects at a working distance beyond the point ofintersection. When the surgical site is reached, he or she may want toobserve objects which are closer to the viewing end of the endoscope orbeyond the point of intersection when the surgical procedure is actuallyperformed.

Additionally, stereo endoscopes are not always precisely opticallyaligned due to, e.g., manufacturing constraints, and the like. In otherwords, the viewing axes sometimes do not intersect either precisely orat all. When such an endoscope is used, without compensating for suchmisalignment of the viewing axes, it could lead to the surgeonexperiencing premature eye strain, headache, and/or general fatigue inattempting to compensate for the imprecise “stereo image”. Compensatingfor such misalignment should increase the time between when a surgeoncommences a surgical procedure and when he or she becomes tired and thusless efficient at performing the procedure. Accordingly, it shouldenhance the overall comfort of the surgeon, enabling the surgicalprocedure to be performed in a more precise manner, by enhancing theoptical comfort experienced by the surgeon during a surgical procedure.

It is an object of this invention to provide an imaging system whichprovides for adjustment of the working distance of a stereo endoscope.It is also an object of this invention to provide an imaging systemwhich provides for alignment of the viewing axes of such a stereoendoscope. Further objects will be apparent from the followingdescription of the preferred embodiments of the invention.

SUMMARY OF THE INVENTION

According to one preferred aspect of the invention, there is provided amethod of imaging a target site with a stereo imaging system. The methodcomprises capturing a right and a left optical image of the target site.It further includes transforming the right and the left optical imagesinto regulatable information, converting the regulatable informationinto opposed images of the target site displayed in a stereo display ofthe stereo imaging system, one of the opposed images being associatedwith the right optical image and the other of the opposed images beingassociated with the left optical image, and regulating the regulatableinformation to cause the position of the target site displayed on theopposed images to change relative to each other.

According to another preferred aspect of the invention, there isprovided a method of aligning opposed images of a stereo imaging system.The method comprises capturing a right and a left optical image of atarget site, transforming the right and the left optical images intodigital information in the form of digital arrays associated with eachof the right and the left optical images, converting the digitalinformation associated with each digital array into opposed images ofthe target site displayed on a stereo display of the stereo imagingsystem, and isolating a portion of at least one digital array so thatonly digital information associated with the isolated portion of thatarray is converted into an associated image, the isolated portion beingselected so as to align the opposed images displayed on the stereodisplay.

According to yet another preferred aspect of the invention, there isprovided a method of adjusting the stereo working distance of a stereoimaging system. The method comprises capturing a right and a leftoptical image of a target site, transforming the right and the leftoptical images into digital information in the form of digital arraysassociated with each of the right and the left optical images,converting the digital information associated with each digital arrayinto opposed images of the target site displayed on a stereo display ofthe stereo imaging system, and isolating a portion of at least onedigital array so that only digital information associated with theisolated portion of that array is converted into an associated image,the isolated portion being selected to cause the working distance tovary.

According to yet a further preferred aspect of the invention, there isprovided a stereo imaging system comprising a stereo image capturedevice for capturing a right and a left optical image of a target site.The system further includes an image transformer operatively associatedwith the image capture device for transforming the right and the leftoptical images into corresponding regulatable information, two displayareas operatively associated with the image transformer for displaying aright and a left image derived from the corresponding regulatableinformation, and a processor arranged to regulate the regulatableinformation to cause the positions of the target site displayed on theopposed images to change relative to each other.

Advantageously, the regulatable information is in the form of digitalinformation.

According to yet a further preferred aspect of the invention, there isprovided a method of imaging a target site with a stereo imaging system.The method comprises capturing first and second optical images of thetarget site as regulatable information, the first and second opticalimages defining a positional relationship, manipulating the regulatableinformation to define an altered positional relationship, and convertingthe manipulated regulatable information into left and right images ofthe target site and displaying the left and right images on a stereodisplay of the stereo imaging system.

According to yet another preferred aspect of the invention, there isprovided a method of aligning opposed images of a stereo imaging system.The method comprises capturing a right and a left optical image of atarget site, transforming the right and the left optical images intoregulatable information associated with each of the right and leftoptical images, converting the regulatable information into images ofthe target site displayed on a stereo display of the stereo imagingsystem, and regulating the regulatable information to align the imagesdisplayed on the stereo display.

According to yet another preferred aspect of the invention, there isprovided a method of adjusting the stereo working distance of a stereoimaging system. The method comprises capturing a right and a leftoptical image of a target site, transforming the right and the leftoptical images into regulatable information associated with each of theright and the left optical images, converting the regulatableinformation into images of the target site displayed on a stereo displayof the stereo imaging system, and regulating the regulatable informationto adjust the stereo working distance of the stereo imaging system.

According to yet another preferred aspect of the invention, there isprovided a method of producing a stereo image of a site at apredetermined position, the method comprising aiming a viewing end of astereo endoscope at the site so that the first image of the site ispassed along a first optical path of the stereo endoscope and a secondimage of the site is passed along another optical path of the stereoendoscope. The method further comprises converting said first and secondimages into corresponding first and second sets of electronicallyreadable information, causing the first set of electronically readableinformation to be transferred into a first visual image on a firstdisplay area, causing the second set of electronically readableinformation to be transferred into a second visual image on a seconddisplay area, and directing the images from the display areas to thepredetermined position so that, at the predetermined position, theimages together form a stereo image viewable by an operator.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of theinvention will be apparent from the following more particulardescription of preferred embodiments and features of the invention andwith reference to the drawings. It will be appreciated that the drawingsare schematic and merely serve to illustrate the principles of theinvention. In the drawings:

FIG. 1 shows a schematic three-dimensional view of one embodiment of animaging system in accordance with the invention;

FIG. 2 shows a schematic three-dimensional view of an object viewing endportion of an endoscope of the system shown in FIG. 1;

FIG. 3 shows a schematic plan view of two Charge Coupled Devices (CCDs)of the system shown in FIGS. 1 and 2, each CCD being operativelyassociated with a separate display monitor;

FIG. 4 shows a schematic representation of an isolating member inaccordance with another aspect of the invention;

FIG. 5 shows a schematic view of another embodiment of an imaging systemin accordance with the invention;

FIG. 6 shows a schematic diagram of part of the system shown in FIG. 5and further indicates steps involved in a method of aligning the viewingaxes of an endoscope of the system, in accordance with the invention;

FIG. 7 shows a schematic plan view of two digital information arrays ofthe system shown in FIG. 5;

FIG. 8 shows a schematic diagram of two images displayed on a stereoviewer assembly of the system shown in FIG. 5, the images having beenderived from the digital information arrays shown in FIG. 7;

FIG. 9 shows a schematic diagram of two images displayed on the stereoviewer assembly, the images having been derived from isolated portionsof the digital information arrays of FIG. 7 to compensate formisalignment in the system;

FIG. 10 shows a schematic diagram of two images displayed on the stereoviewer assembly, the images being associated with a target used toassist in compensating for misalignment in the system;

FIG. 11 shows a schematic three-dimensional view of a target device usedto assist in compensating for the misalignment in the system;

FIG. 12 shows a plan view of the target device along arrow X in FIG. 11and shows the target viewed in the images shown in FIG. 10;

FIG. 13 shows a digital array of one of a right and left optical image;

FIG. 14 shows the digital array of FIG. 13, and further shows a leastsquares fit to columns and rows of the digital array, where the leastsquares fit has been computed based on the mean between the locations ofa first and a last thresholded maximum value for each row and column;

FIG. 15 shows a graphical representation of digital values in the rowsand columns of FIGS. 13 and 14; and

FIG. 16 shows a position of an intersection between the least squaresfit for the rows and columns, superimposed on the graphicalrepresentation of FIG. 15.

DETAILED OF THE SPECIFIC EMBODIMENTS

Referring to the drawings, and with specific reference to FIG. 1, animaging system, in accordance with the invention, is generally indicatedby reference numeral 10.

While the preferred embodiments of the present invention are describedas including converting the various optical images into regulatableinformation in the form of digital information, other forms ofregulatable information could also be used, such as, for example,analogue information from a video and a sync signal, as will be apparentto one of ordinary skill in the art upon reading this disclosure.

Although throughout the rest of this specification the invention will bedescribed with reference to its application in a minimally invasivesurgical apparatus employing an image capturing device in the form of anendoscope, it is to be understood that the field of the invention is notnecessarily limited to this application. Accordingly, the invention canbe used in, e.g., open surgery, also.

The system 10 includes a stereo imaging device in the form of a stereoendoscope 12, for example. The system 10 further includes two ChargeCoupled Devices (CCDs) 13 and 14, optical lenses 16 and 18, and readmeans 20, 22 for reading the CCDs and converting information read on theCCDs into a digital format. The read means 20, 22 is typically anappropriate electronically driven system such as a Camera Control Unit(CCU) that transforms optical information read by the CCDs 20, 22 intodigital format. The CCD and CCU arrangements can be of the typeavailable from Panasonic™ under the part nos.: GP-US522/GP-US532 3CCDcolor CCU. Accordingly, an electronic processor (not shown) is typicallyin operative communication with the read means 20, 22. The system 10 yetfurther includes two display stations, e.g., in the form of Cathode RayTubes (CRTs) schematically indicated at 24, 26, each of which isoperatively connected to one of the read means 20, 22 as indicated bylines 28, 30 so as to convert preferably digitally formatted informationreceived from the read means 20, 22 into corresponding visual images onthe CRTs 24, 26. It is to be appreciated that although reference is madeto Cathode Ray Tubes, any other appropriate display station or visualscreening apparatus can be used, e.g., a liquid crystal display, or thelike. An assembly of reflecting surfaces or a reflective train, e.g.,comprising mirrors, or the like, is generally indicated by referencenumeral 32. The assembly 32 conveys the images from the CRTs 24, 26 to aviewer at 34. It will be appreciated that the direction of view at 34 isnormal to the page, a right eye image being indicated at 34.1, and aleft eye image being indicated at 34.2 for example.

As can best be seen with reference to FIG. 2, in which the samereference numerals as used in FIG. 1 represent the same parts unlessotherwise stated, the stereo endoscope 12 includes two optical lenses36, 38. The optical lenses 36, 38 have viewing axes as indicated bylines 40, 42. The lenses 36, 38 are fixed in position so that the axes40, 42 intersect at 44. If an object is viewed at a working distancegenerally corresponding to the intersection 44, namely on a plane at A,the object is viewed generally at an optically correct working distance.The further the object is removed from the intersection 44, the moreoptically incorrect the object is viewed. Accordingly, should the objectto generally at another plane relative to the endoscope 12, such as theplane indicated at B, the object would not be viewed optically correctlythrough the endoscope 12 since the object is not at the working distanceA of the endoscope, namely at the intersection 44. Furthermore, thegreater the positional deviation of the object sought to be viewedthrough the endoscope 12 relative to the intersection 44, the moreoptically incorrect the viewed stereo image becomes.

As mentioned earlier, it is often required to view an object at varyingdistances relative to an object viewing end 12.1 of the endoscope 12.For example, during endoscopic surgery, a surgeon often desires to viewobjects beyond A when introducing the endoscope 12 into a patient's bodyso as to locate a predetermined surgical site within the body and thenmay need to view objects at other working distances, which can bepositioned closer to the viewing end 12.1 than the distance A, or beyondthe intersection corresponding to working distance A, when the surgicalsite is located and the surgical procedure is to be performed. Thesystem 10 is arranged to adjust the working distances of the endoscope12, i.e., by effectively changing the distance of the intersection 44relative to the object viewing end 12.1 of the endoscope 12.

As can best be seen with reference to FIG. 2, a viewing area, or thefield of view, of each lens 36, 38 can be represented by a sectionthrough an imaginary conical volume indicated by referenced numerals36.1, 38.1. If an object is on a plane at A, as indicated at A1, withinthe bounds of the optical field of view, as indicated by the oval shapedareas 60, 62, the object is viewed at an optically correct workingdistance. It will be appreciated that the axes 40, 42 are positioned atthe centers of the oval shaped areas 60, 62 in an optically alignedsystem. The areas 60, 62 are illustrated as being coincident. However,in the case of an optically misaligned stereo endoscope, the areas wouldnot be coincident, and the axes would not intersect at 44, but would beoffset relative to each other.

Images of an object viewed at A are passed through the lenses 36, 38,along optical paths indicated by arrows 44, 46 in the endoscope 12, andare then magnified through lenses 16, 18 and are then projected ontooptically sensitive surfaces of e.g., the CCDs 13, 14, as indicated byarrows 44A, 46A in FIG. 1.

As can best be seen with reference to FIG. 3, the CCDs 13, 14 eachinclude an array of photosensitive pixels schematically indicated by thedots 50, 52. The images from the lenses 16, 18, are projected onto thephotosensitive surfaces of the CCDs 13, 14, respectively. Apredetermined portion of the images represented by cropping windows 54,56 is read by the read means 20, 22. Only a portion of each image,namely the cropping windows 54, 56, is read, so as to avoid peripheraldistortions at the outer region of the viewing areas 60, 62. The readmeans 20, 22 reads the information from the photosensitive surfaces ofthe CCDs 13, 14, converts the information into preferably a digitalformat and transmits the digital information to the CRTs 24,26 so as tocause the information to form corresponding images on the CRTs 24,26.

A method employed by the system 10 to vary the working distance of theendoscope includes causing the read means 20, 22 to read only a selectedor isolated portion of the cropping windows 54, 56 along an X axis ofthe coordinate frame indicated at 17. Thus, by way of example, should itbe required to increase the working distance beyond of the intersection44 of the endoscope 12 the read means 20, 22 is caused to read innerportions of the cropping windows 54, 56, namely portions 54.1, 56.1only, as can best be seen in FIG. 3. It will be seen that the distancebetween the centers 54.2, 56.2 of the cropping windows 54.1, 56.1 arecloser to each other along the X axis than is the case with the centers54.3 and 56.3 of the cropping windows 54, 56. The centers 54.2, 56.2,54.3, and 56.3 determine the eventual viewing axes. Thus, where the fullcropping windows 54, 56 are read, the centers 54.3, 56.3 correspond toviewing axes 40, 42. Where cropping windows 54.1, 56.1 are read, thecenters 54.2, 56.2 correspond with viewing axes which intersect at apoint further away from the viewing end 12.1 of the endoscope 12 thanthe intersection 44 and, for example, viewing axes 70, 72 whichintersect at 73 on a plane B, with reference to FIG. 2 of the drawings,can be achieved. It will be appreciated that whether the viewing axescorresponding with the centers 54.2, 56.2 of the cropping windows 54.1,56.1 intersect at a point further away from or closer to the viewing end12.1 depends on the system configuration e.g. the optic lensarrangement, used in the endoscope. Naturally, should portions of thecropping windows 54, 56 be selected which define centers further removedfrom each other along the X axis, the opposite is true. In this way, anobject at a working distance differing from the working distance set bythe lenses 36, 38, for example on the plane B, can be viewed in a moreoptically correct manner, by varying the working distance ashereinbefore described.

It will be appreciated that the selected or isolated portions of thecropping windows 54, 56 are converted into corresponding visual imageson the CRTs 24, 26. The smaller the portion of the cropping windows 54,56 selected, the smaller the amount of pixel information defining thevisual images displayed on the CRTs 24, 26 becomes, which negativelyeffects picture clarity. Thus, to preserve picture clarity the selectedcropping window portions should not be too small. This can best beillustrated with reference to FIG. 3. In FIG. 3, it will be seen thatthe number of pixels 50, 52 in the cropping windows 54, 56 is greaterthan that in the cropping windows 54.1, 54.2. Thus the resultant imageson the CRTs 24, 26 are then derived from less information, whichnegatively effects picture quality. Accordingly, a stage is reachedwhere increased variation of the working distance and correspondingsacrifice of picture clarity is not viable.

In the imaging system 10, isolating portions of the cropping areas 54,56 as described above is used also in a method of aligning the imagesdisplayed on the CRTs 24, 26 in the case of misalignment. In theendoscope 12, as already mentioned, it can happen that e.g., the lenses36, 38, are not perfectly aligned. This can occur as a result ofmanufacturing constraints, or the like. In such a case, the axes 40, 42may not intersect but can be marginally offset relative to each other onan X-Y plane. To compensate for misalignment of the axes 40, 42 inaccordance with the method of the invention, a target in the form of across, for example, is positioned at a predetermined distance from theviewing end 12.1 of the endoscope 12, such as at the working distance Aof the endoscope 12. This can be achieved by a suitable set piece ortarget device (not shown) releasably locatable on the viewing end 12.1of the endoscope 12 so that the target or cross is at a specificdistance relative to the end 12.1 of the endoscope 12. The images passedalong the respective optical paths 44, 46 are typically colored so thatthe image passed along path 20 is green, for example, and the imagepassed along path 22 is magenta, for example. In the case ofmisalignment, the crosses viewed through the viewer at 34 would not besuperimposed. It will be appreciated that distinctively coloring theimages passing along the two optical paths is to assist in the aligningof the crosses displayed on the CRTs 24, 26 during alignment. Portionsof the cropping areas 54, 56 are then selected to cause their centers tobe moved along the X and Y axes until the crosses viewed through theviewer 34 are superimposed or in register. In such a condition themisalignment is compensated for. The area of the cropping windows arechosen to be as large as possible while maintaining the registration ofthe crosses as viewed to cause the images displayed at the viewer 34 tobe derived from as much pixel information as possible, thereby tooptimize picture quality. The chosen portions of the cropping windowsare then set by the system 10 so that alignment is retained in use. Theworking distance can then selectively be varied by isolating croppingareas having centers at different spacings relative to each other alongthe X axis.

It has been found that misalignment of the axes 40, 42 can cause a user,e.g., a surgeon, to experience discomfort, e.g., eye strain, after acertain length of use. By compensating for misalignment as describedabove, the length of time before such discomfort is experienced istypically increased.

In accordance with another aspect of the invention, the endoscope system10 includes means for focusing the resultant images on the CRTs.Focusing is achieved by mechanical means arranged to vary the relativedistances between the lenses 16, 18 and their corresponding CCDs 13, 14.The mechanical means may be in any appropriate form. For example, themechanical means may include a foot pedal, the surgeon adjusting focusby varying the inclination or degree of depression of such a foot pedal.Instead, or in addition, manually operable switches can be provided on aconsole or the like. The mechanical means can be motorized. An exampleof such focusing means is described hereinbelow with reference to FIG. 5of the drawings.

It will be appreciated that in a surgical environment sterility is ofcardinal importance. Since the endoscope is used in a surgicalprocedure, it should be sterilizable. It would be advantageous if therest of the imaging system could be isolated from the endoscope toobviate having to sterilize the rest of the imaging system, and topermit the endoscope to be changed during a surgical procedure withoutcontaminating the sterile environment.

In FIG. 4, a forward or operative end portion of the imaging system 10is generally indicated by reference numeral 90. The portion 90 includesthe endoscope 12. The endoscope 12 is normally connected to a videocamera arrangement or camera head including the CCDs and CCUs. Part ofthe camera head is schematically indicated by reference numeral 92. Theimaging system 10 includes an isolating member 94. One end portion 94.1of the member 94 is releasably couplable on an operative rear end 12.2of the endoscope 12. This can be achieved by any appropriatecomplementary coupling formation on the rear end 12.2 and the endportion 94.1. The complementary coupling formations can be in the formof mating internal and external screw-threads on the end 12.2 andportion 94.1. An opposed end portion 94.2 of the member 94 is releasablycouplable to the camera head 92. This can also be achieved by anyappropriate complementary coupling formations.

The isolating member 94 can include an optically correct window topermit optical information to pass therethrough from the endoscope 12 tothe camera head 92. A surgical drape is connectable around or onto aperiphery 94.3 of the member 94. To this end the member 94 can includelocation means, e.g., a suitable peripheral surface, or the like, toenable adhesive tape to be adhesively secured thereon and to which asurgical drape can be releasably adhered. Alternatively, the member 94can form an integral part of a surgical drape.

Accordingly, the endoscope system 10 includes a window member mountableat the end of the endoscope 12 remote from the lenses 36, 38. In use, adrape is located over the rest of the imaging system 10, and tapedaround the periphery 94.3 of the member 94. In this manner, the rest ofthe imaging system is isolated from a surgical procedure and accordinglyneed not be sterilized between procedures. Furthermore, since the drapeand window assembly are sterile, any sterile objects that contact thedrape or window assembly will remain sterile. This permits endoscopes tobe changed repeatedly during the procedure without requiring a sterilecamera head.

Referring now to FIG. 5 of the drawings, a preferred imaging system inaccordance with the invention is generally indicated by referencenumeral 110. The system 110 includes an image capturing device in theform of a stereo endoscope 112 having a viewing end 112A. The endoscope112 is operatively connected to a camera head 114 by means of a coupler116. The coupler 116 is arranged to enable the rest of the imagingsystem 110 to be isolated from the endoscope 112, as described abovewith reference to FIG. 4 of the drawings. Isolation of the rest ofsystem 110 is typically achieved by means of a surgical drape positionedto cloak the rest of the system 110, the drape being secured to thecoupler 116.

The camera head 114 typically includes two focusing lenses 118, 120 forfocusing a right and left optical image relayed along optical paths 122,124, respectively. The camera head 114 further includes two ChargeCoupled Devices 126, 128 arranged to cooperate with the optical paths122, 124. The right and left optical images are typically directedthrough the lenses 118, 120 onto photosensitive surfaces defined by theCCDs 126, 128. Camera Control Units 130, 132 are operatively connectedto the CCDs 126, 128 so as to scan the photosensitive surfaces of theCCDs and to convert information derived from the scanned photosensitivesurfaces into video format.

Video formatted information associated with the right optical image isthen forwarded to a video processor 134 as indicated by line 134A.Similarly, video formatted information associated with the left opticalimage is forwarded to a video processor 136 as indicated by line 136A.The video processors 134, 136 process the video formatted information toenhance the information by e.g., decreasing electronic noise, enhancingedges to provide sharper or crisper images, and/or the like.

After the video formatted information associated with the right and theleft optical images has been processed by the processors 134, 136, theinformation is forwarded to another video processor indicated byreference numeral 138. The information associated with the right and theleft optical images is fed to the video processor 138 as indicated bylines 139A, 139B, respectively, and is typically in the form of digitalinformation.

The video processor 138 includes an alignment and working distancecompensation processing stage indicated by reference numeral 140. In thestage 140, the digital information sets associated with the right andthe left optical images are each arranged in the form of a digitalinformation array as indicated schematically in FIG. 7 of the drawings,and as described in greater detail hereinbelow. To facilitate alignmentof the right and the left images and working distance adjustment of theendoscope 112 the digital information is regulated as describedhereinbelow.

After the stage at 140, the regulated digital information is scanned ata scan conversion stage 142 to procure associated analog informationdisplayable on e.g., Cathode Ray Tube monitors. After the scanning stage142, the analog information associated with the right optical image isforwarded to a monitor 144 and the analog information associated withthe left optical image is forwarded to a monitor 146, as indicated bylines 144A, 146A, respectively. At the monitors 144, 146, theinformation forwarded along lines 144A, 146A is used to form opposedimages each of which is displayed on a monitor display 144B, 146B of themonitors 144, 146. The images displayed on the monitor displays 144B,146B are then guided along optical paths 144C, 146C to a viewer 148. Theviewer 148 typically includes two opposed reflective surfaces 148A, 148Bfor directing the optical paths 144C, 146C in generally paralleldirections to be observed separately by each of an observer's eyes. Itwill be appreciated that the observer's eyes are positioned across aninterface 148C between the reflective surfaces 148A, 148B to cause theimage associated with the right optical image to be observed by theobserver's right eye and the image associated with the left opticalimage to be observed by the observer's left eye. It will be appreciatedthat the direction of view in FIG. 5 of an observer is normal to thepage. Naturally it will be appreciated that FIG. 5 is a schematicdiagram and that the viewer 148 can be positioned in any appropriatemanner relative to the monitors 144, 146 so that the direction of viewof an observer need not to be normal to the page as indicated, but canbe in a direction toward the monitors 144, 146 for example. Furthermore,the monitors 144, 146 can be positioned at any appropriate positionrelative to each other and the rest of the system, in which case theoptical paths 144C, 146C can be tailored to direct the images from themonitors to the viewer.

It would be advantageous to present an optically correct stereo image tothe observer. To approach such an optically correct stereo image, itwould be advantageous if the system 110 is enabled to compensate forcertain optical inaccuracies or shortcomings which may be present in thesystem. One such shortcoming could arise due to optical misalignmentbetween the optical paths 122, 124 downstream of the CCDs 126, 128. Suchmisalignment can arise due to lens misalignment, for example. Thepreferred system 110 provides means for compensating for suchmisalignment which will now be described with reference to FIGS. 7 to 9of the drawings.

Referring initially to FIG. 7 of the drawings, a digital informationarray associated with the right optical image is generally indicated byreference numeral 150 and a digital information array associated withthe left optical image is generally indicated by reference numeral 160.As mentioned hereinbefore, the digital information arrays 150, 160 werederived at the stage 140 as indicated in FIG. 5 of the drawings. It willbe appreciated that the images displayed on the monitors 144, 146, shownin FIG. 5 of the drawings, are derived from the digital informationcontained in the arrays 150, 160. Referring to FIG. 5 of the drawings,means of the system 110 to compensate for misalignment includes acontroller 149, which cooperates with the arrays 150, 160. Thecontroller 149 cooperating with the arrays 150, 160 enables discreteportions of the arrays 150, 160 to be isolated so that the imagesdisplayed on the monitors 144, 146 are derived from the isolatedportions of the arrays 150, 160 only.

It will be appreciated that in an optically aligned system, a targetsite indicated at T in FIG. 5 of the drawings would be transformed intoassociated digital information as indicated by the crosses at TR, TL inthe digital arrays 150, 160 in FIG. 7. If all the digital information ofthe arrays 150, 160 is then used to form the images on the displays144B, 146B of the monitors 144, 146, then the target site T would bepositioned generally at centrally disposed positions in the imagesdisplayed on the monitors 144, 146 as indicated by TR1, TL1 in FIG. 8 ofthe drawings. In the case of optical misalignment, the informationassociated with the target site T on the arrays 150, 160 would be offsetas indicated, by way of example, by the dashed crosses TR* and TL* inFIG. 7. Naturally, if all the information of the arrays 150, 160 is thenused to form the images on the monitors 144, 146 the target site T wouldappear at centrally offset positions indicated at TR1* TL1* in FIG. 8,for example.

By enabling the system 110 to isolate portions of the arrays 150, 160and to cause only such portions to define the information from which theimages on the monitors 144, 146 are derived, the misalignment can becompensated for. Accordingly, referring to FIG. 7 of the drawings, giventhe misalignment as indicated at TR*, TL* described above, the images ofthe monitors 144, 146 can be aligned by selecting and isolating theportions in dashed lines indicated by reference numerals 142, 162,respectively, and using only this information to derive the imagesdisplayed on the monitors 144, 146. In such a case, the digitalinformation relating to the target site T, in the case of misalignment,namely at TR₁*, TL₁*, would then be imaged at centrally disposedpositions on the monitors 144, 146 as indicated by TR₂*, TL₂* in FIG. 9of the drawings. In this way, the misalignment of the optical paths 122,124 is compensated for. It will be appreciated that not onlymisalignment which can arise in the optical paths 122, 124 can becompensated for in this manner, but misalignment elsewhere also, such asbetween the CCDs 126, 128 and the CCUs 130, 132, for example.

To assist in the alignment process described above, use is typicallymade of a target device schematically indicated at 121 in FIG. 5 of thedrawings. The target device is shown schematically in greater detail,and on an enlarged scale, in FIGS. 11 and 12 of the drawings. The targetdevice 121 is in the form of a cap releasably mountable on the viewingend 112A of the endoscope 112 as indicated in FIG. 5 of the drawings.The target device includes a body 121.1. The body 121.1 defines aninternal seat 121.2 extending along an internal periphery 121.3, wherebyit can be mounted on the viewing end of the endoscope by positioning theviewing end 112A of the endoscope 112 in the seat 121.2 therebyfrictionally to engage over said end 112A.

The target device 121 further includes a target 121.4 opposed from theseat 121.2. The target 121.4 is at a fixed distance W from the seat121.2. The target 121.4 is shaped in the form of a cross. However, itwill be appreciated that any suitably shaped target can be used instead,e.g., a centrally disposed circle, or the like.

To assist the alignment process, and as represented in FIG. 6 of thedrawings, outputs from the video processor 138 as indicated generally at170A, 170B which are normally operatively connected to inputs 172A, 172Bassociated with the monitors 144, 146, are interchanged so as to causeeach image displayed on the monitors 144, 146 to be a mix of theinformation associated with both paths 122, 124. The outputs at 170A,170B are associated with conventional red, green, and blue signals asindicated by the letters R, G, B and R₁, G₁, B₁, respectively. Theoutputs 170A, 170B are normally operatively connected to the inputs172A, 172B as indicated by the solid parallel lines. To interchange theoutputs, the R₁ signal is routed to the R input, the B₁ signal is routedto the B input and the G signal is routed to the G₁ input. Accordingly,in this manner, the image displayed on each viewer 144, 146 is in theform of a composite image associated with both optical images 122, 124.Conveniently, to enable each image of the composite image to be readilydistinguished, the images are caused to have different distinct colors,e.g., green and magenta, or the like.

When the target device is mounted on the viewing end 112A of theendoscope 112 and the outputs 170A, 170B and the inputs 172A, 172B areinterchanged in this fashion, the target, i.e., the cross, in the caseof misalignment, is displayed in a misaligned manner on each monitor144, 146 as indicated schematically by the solid lines 111A and dashedlines 111B, respectively, in FIG. 10 of the drawings. The controllershown in FIG. 5 is then used to select portions of the arrays 150, 160which would cause the targets as viewed on the monitors 144, 146 to besuperimposed or in register. This is typically achieved by the observerwhile viewing the images at the viewer 148 and manipulating an input 151operatively linked to the controller 169 as indicated in FIG. 5. Theinput can be in any appropriate form such as opposed vertical andhorizontal switches, foot pedals, voice commands, or the like.Typically, opposed vertical and horizontal switches are used so thatwhen an upper of the vertical switches is depressed, a center of aportion of the one digital array 150, 160 is caused to moveprogressively upwardly and a center of a portion of the other digitalarray 150, 160 is caused to move progressively downwardly. Similarly ifthe lower vertical switch is depressed, the center of the portionsselected move progressively in directions opposite to the directions ofthe movement when the upper vertical switch is depressed. Naturally,when the one horizontal switch is depressed, centers of the portionsselected move progressively toward each other, and when the otherhorizontal switch is depressed, the center of the portions selected moveprogressively laterally away from each other. The switches are operatedin this manner until the images of the composite image on the viewers144, 146 are superimposed or in register. The system 110 is then set sothat the selected isolated portions of the arrays 150, 160 which causethe images to be aligned are retained thereby to compensate formisalignment.

It will be appreciated that instead of selecting portions of the arrays150, 160, the arrays 150, 160 can be arranged to form parts of largerarrays (not shown). Alignment can then be established in similarfashion, but by moving the arrays 150, 160 as a whole relative to thelarger arrays. It will further be appreciated that the controller 149can be arranged to select and isolate a portion of only one of thearrays 150, 160 to establish alignment. It will yet further beappreciated that once alignment has been achieved, conventionalconnection between outputs 170A, 170B and corresponding inputs 172A,172B is re-established.

Once misalignment has been compensated for, working distance variationis achieved in similar fashion, except that portions of the digitalarrays 150, 160 are selected and isolated which have centers closer toeach other in an X direction as indicated by the coordinate referenceframe 161 in FIG. 7 of the drawings, to increase the working distanceand portions having centers further removed in an X direction areisolated to decrease the working distance. It will be appreciated thatduring the misalignment compensation procedure described above theisolated portions can vary in an X and Y direction whereas oncealignment is achieved, working distance variation is achieved byisolating portions such that the distance between their centers variesalong the X axis only. Furthermore, when misalignment compensation isachieved using the target device 121, the working distance is typicallyat a distance corresponding to the distance between the target cross andthe viewing end 112A of the endoscope. Thereafter, varying the workingdistance can typically be achieved by a suitable input indicated at 153in FIG. 5 of the drawings. The suitable input can be in any appropriateform such as a hand operable switch, depressible foot pedal, or thelike. Typically, the input can be arranged to cause stepwise adjustmentof preset working distances. Accordingly, a plurality, e.g., three,settings can be provided to vary the working distance between threepreset distances.

Referring now to FIGS. 13 to 16 of the drawings, another method ofaligning opposed images of a stereo imaging system will now bedescribed.

Referring initially to FIG. 13 of the drawings, a digital array isgenerally indicated by reference numeral 110. It will be appreciatedthat the digital array 110, is shown to be a 20×20 array forillustrative purposes only. Typically, a 64×64 array can be used. Thearray 110 includes a plurality of rows 112. It further includes aplurality of columns 114. The rows 112 and columns 114 define discretelocations 116 in which numerical values relating to pixel information ofa captured image is contained. For illustrative purposes only, a seriesof elliptical rings 118 is shown superimposed on the array 110. Theseries of elliptical rings 118 represent a target viewed by theendoscope and from which the digital information in the array 110 wasderived. Toward a central position of the elliptical rings the numericalvalues contained in the rows 112 and columns 114 are of higher numericalvalue than in the peripheral regions. Accordingly, the target viewed bythe endoscope has a brighter portion corresponding to the centrallydisposed position of the elliptical rings 118 than at a position awayfrom the centrally disposed position.

It will be appreciated that although, for the sake of example, referenceis made to elliptical rings 118, the target as described with referenceto FIGS. 11 and 12 may be used instead. It will further be appreciatedthat the digital array 110 is only one digital array corresponding toeither the left or the right optical image captured by the endoscope.Accordingly, when a target site is viewed by the endoscope, a digitalarray corresponding to digital array 110 would be captured for each ofthe left and the right optical images.

When the digital arrays for the left and the right optical images havebeen captured in this fashion, any misalignment in the system can becompensated for automatically. A method for automatically compensatingfor misalignment will now be described in further detail. Once thedigital arrays have been captured a processor analyzes each row 112 todetermine the highest numerical value in that row. Once the highestnumerical value in a particular row has thus been determined, thatmaximum value is multiplied by a predetermined constant. Typically, thepredetermined constant can have a value ranging between about 0.7 and0.95. A value often used is 0.9. Once the maximum numerical value hasbeen multiplied by the predetermined constant a threshold value for thatrow is defined. The processor then scans the row and determines thelocation of the first numerical value which exceeds the threshold valueas well as the location of the last numerical value which exceeded thethreshold value. In this manner, two opposed locations are determined inthe row. The processor then determines the mean location between theselocations.

The above procedure is repeated for each row 112. Thus, a location ineach row 112 is determined which corresponds to the mean between thefirst and the last value which exceeded the threshold value for eachspecific row. The same method is employed for each column 114.Accordingly, a location in each column is determined which representsthe mean between the locations of the first and the last numericalvalues in that column which exceeded the threshold value determined forthat column.

Referring now to FIG. 14, the mean values determined as described abovefor each row 112 is indicated by the crosses 120. Similarly, the meanvalues for each column is indicated by the circles 122. The processorthen computes a least squares fit using the locations of the meanvalues. The results of this computation is indicated by the straightlines 124 and 126, respectively. The location of the intersectionbetween these lines, indicated at 128, is then determined.

It will be appreciated that the location of the mean value for everysingle row and column need not necessarily be determined. Instead, onlythe locations of the mean values of predetermined spaced-apart rows andcolumns can be determined. Accordingly, the location of the mean valueof every second, third, fourth or the like, row and column can bedetermined to provide adequate data to determine the intersections 128for each digital array.

Referring to FIG. 15, a graphical representation of the numerical valuesin the rows 112 and columns 114, is diagrammatically indicated in athree dimensional format. The numerical values are indicated at 130.When the computations as described above have been completed, theintersection 128 indicated in FIG. 14 corresponds generally to theposition within the digital array 110 corresponding to a maximumnumerical value. This can best be seen with reference to FIG. 16 inwhich the lines 124, 126 have been superimposed on the diagram of FIG.15.

Once the locations of the intersection between lines 124 and 126 havebeen determined in the digital array corresponding to each of the leftand the right optical images, the processor selects portions or croppingwindows of each of the digital arrays such that the location of theintersections 128 for each array occupy the same position relative tothe selected portions or cropping windows. Only digital informationrelating to the selected portions is then used to display an image onthe viewer corresponding to the left and the right optical imagesrespectively to align the images.

It will be appreciated that to compensate for misalignment as describedabove, use can be made of an appropriate target device as describedearlier in this specification. However, as those skill in the art willbe aware, compensation for misalignment using this method can beaccomplished without use of a target device. In such a case, where useis not made of a target device, the endoscope can be directed to captureany appropriate image, such as of the surgical site on which it isintended to perform a surgical procedure, and use can then be made ofappropriate pattern matching templates, or the like, to determineintersections as described above for the digital arrays corresponding tothe image of the surgical site. Furthermore, it will be appreciated thatthe system can be arranged automatically to compensate for misalignmentas described above periodically. Furthermore, it is to be appreciatedthat the compensation for misalignment when performed periodically canbe arranged to compensate for vertical alignment between the imagesonly, so that if the surgeon has selected a particular working distance,that working distance is maintained.

Referring again to FIG. 5 of the drawings, the system 110 is providedwith a focusing arrangement generally indicated by reference numeral180. The arrangement 180 is arranged to move the lenses 118, 120selectively in the direction indicated by arrows 182. This is achievedby means of an electric motor 184 operatively associated with the lenses118, 120 through an appropriate transmission, e.g., gears, or cable andpulley arrangements, or rack and pinion arrangements, or the like. Themotor 184 is driven by means of a focus controller 186. The focuscontroller is typically connected to an input device at the operatorconsole. The input device can be in any appropriate form, e.g.,switches, knobs, voice control, or the like. Accordingly, an operatorcan adjust the focus by means of the input device.

While exemplary preferred embodiments have been described in somedetail, for purposes of clarity of understanding and by way of exampleonly, a variety of changes, modifications, and adaptations will beobvious to those skilled in this art. Accordingly, the scope of thepresent invention is reflected by the appended claims. Thus, while thepreferred embodiments of the present invention are described asincluding converting the various optical images into regulatableinformation in the form of digital information, other forms ofregulatable information, such as analog information, could be usedinstead, as will be apparent to one of ordinary skill in the art uponreading this disclosure. For example, to vary the position of the imagesdisplayed on the monitors to vary the working distance, regulatableinformation including a sync signal and a video signal for input to therespective monitors can be regulated so as to skew the sync and videosignals relative to each other and a blanking system of the monitors canbe permitted to select active pixels to provide such working distancevariation or compensation for misalignment.

1-62. (canceled)
 63. A method for virtually varying a working distanceof a stereoscopic imaging device for displaying portions of capturedleft and right images on a three-dimensional viewer, comprising:receiving information of captured right and left images from thestereoscopic imaging device; receiving a command to vary a workingdistance associated with the stereoscopic imaging device; processing theinformation of the captured right and left images by cropping each ofthe left and right images so as to virtually vary the working distanceaccording to the command; and providing the cropped right and leftimages to the three-dimensional viewer for display on thethree-dimensional viewer.
 64. The method of claim 63, wherein theprocessing of the information of the captured right and left imagescomprises: determining whether the command to vary the working distanceindicates that the working distance is to be increased or decreased; ifthe working distance is to be increased, then cropping the left andright images so that left image reference points in the uncropped leftimage appear left of center in the cropped left image and right imagereference points in the uncropped right image appear right of center inthe cropped right image; and if the working distance is to be decreased,then cropping the left and right images so that left image referencepoints in the uncropped left image appear right of center in the croppedleft image and right image reference points in the uncropped right imageappear left of center in the cropped right image.
 65. The method ofclaim 63, wherein the command to vary the working distance is receivedfrom a user operable switch.
 66. The method of claim 63, wherein thecommand to vary the working distance is received from a user depressiblefoot pedal.
 67. The method of claim 63, wherein the command to vary theworking distance indicates a stepwise adjustment of present workingdistances.
 68. The method of claim 63, wherein the stereoscopic imagingdevice is a stereoscopic endoscope.
 69. A stereoscopic imaging systemcomprising: a stereoscopic imaging device capturing left and rightimages of a scene; a user operable input device; a three-dimensionalviewer; and a processor configured to generate cropped left and rightimages by cropping the captured left and right images so as to virtuallychange a working distance associated with the stereoscopic imagingdevice according to a command received from the input device, andprovide the cropped left and right images to the three-dimensionalviewer for displaying the scene relative to the virtually changedworking distance.
 70. The stereoscopic imaging system of claim 69,wherein the user operable input device is a hand operable switch. 71.The stereoscopic imaging system of claim 69, wherein the user operableinput device is a depressible foot pedal.
 72. The stereoscopic imagingsystem of claim 69, wherein the user operable input device is configuredto provide commands indicative of stepwise adjustment of preset workingdistances.
 73. The stereoscopic imaging system of claim 69, wherein theprocessor is configured to: determine whether the command received fromthe input device indicates that the working distance is to be increasedor decreased; if the working distance is to be increased, then crop theleft and right images so that left image reference points in theuncropped left image appear left of center in the cropped left image andright image reference points in the uncropped right image appear rightof center in the cropped right image; and if the working distance is tobe decreased, then crop the left and right images so that left imagereference points in the uncropped left image appear right of center inthe cropped left image and right image reference points in the uncroppedright image appear left of center in the cropped right image.
 74. Thestereoscopic imaging system of claim 69, wherein the stereoscopicimaging device is a stereoscopic endoscope.